GCNU TR 1999 – 002 Linear

This note gives a closed form expression for the linear transform computed by an optimally trained linear heteroencoder network of arbitrary topology trained to minimize squared error. The transform can be thought of as a restricted rank version of the basic linear least-squares regression (discrete Wiener filter) between input and output. The rank restriction is set by the “bottleneck” size of the network – the minimum number of hidden units in any layer. A special case of this expression is the well known result that linear autoencoders with a bottleneck of size r perform a transform equivalent to projecting into the subspace spanned by the first r principal components of the data. This result eliminates the need to explicitly train linear heteroencoder networks. Linear Heteroencoders Sam Roweis Gatsby Unit Carlos Brody Computation and Neural Systems California Institute of Technology 1 Linear autoencoders Because they are fast to train and require few parameters, linear networks1 provide an important performance comparison with more complex data analysis methods. The equivalent linear transform computed by a linear autoencoder networkwith a bottleneck of size r has long been known to be the projection into the subspace spanned by the first r principal components of the training data. This is true if the training algorithm minimizes squared error at the output and achieves the global minimum of that error. Furthermore, Bourlard and Kamp (1988) have shown that if all layers of the network after the bottleneck are linear the optimal transformation remains unchanged even if nonlinear transfer functions are added to units before the bottleneck. In other words, the result still holds for networks that are merely output-linear. These results are important because they eliminate the need to explicitly train linear or output-linear autoencoders. Algorithms such as the singular-value decomposition can be used to quickly compute the optimal transform. These algorithms run in a known fixed time, results are easily reproducible, they are guaranteed to achieve the global minimum of error, and they produce an ordered set of orthogonal eigenvectors. In this note, we present the analogous results for linear (and output-linear) heteroencoder networkswith a bottleneck. Enforcing such a bottleneckmay be important in cases where high dimensionality of the input space leads to overtraining and poor generalization. The results below allow the equivalent transform (and thus performance) of an optimally trained network to be easily computed without explicit training. Many previous authors [Baldi and Hornik, 1989, Kung and Diamantaras, 1991] [Diamantaras and Kung, 1994, Scharf, 1991, Stoica and Viberg, 1996, Ghahramani, 1996] have proved portions of the results we review below. However, the proofs are often part of a more detailed or lengthy discussion and as a result are sometimes mathematically more complex. The goal of this note is to provide a short and simple exposition of these results along with practical expressions for their implementation. y Based on a May 1997 version. During writing, the authors were supported in part by the Center for Neuromorphic Systems Engineering as a part of the National Science Foundation Engineering Research Center Program under grant EEC-9402726 and by the Natural Sciences and Engineering Research Council of Canada under an NSERC 1967 Award. We refer to any network in which the transfer function of every unit is linear as a linear network. Similarly we use the term linear layer for layers in which every unit’s transfer function is a linear function. A network has a bottleneck of size r if no layer has fewer than r units. A network has no bottleneck if all layers have at least as many units as both the input dimension and output dimension. If all layers after the bottleneck layer (or after the last bottleneck if there are several) are linear, we call the network output-linear.